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FIRST EMBODIMENT FIG. 2 




PERFORM FIRST PHOTOGRAPHING AND STORE , ^ 
IMAGE DATA IN MEMORY GROUP 



* 

SHIFT IMAGE SENSING DEVICE BY ONE 
PIXEL IN LONGITUDINAL DIRECTION 



* , 

PERFORM SECOND PHOTOGRAPHING, WITH 
IMAGE SENSING DEVICE SHIFTED BY ONE PIXEL , ^ 
IN LONGITUDINAL DIRECTION, AND STORE 
IMAGE DATA IN MEMORY GROUP 



COMBINE FIRST AND SECOND IMAGES 
STORED IN MEMORY GROUP 



CALCULATE PIXEL SHIFT EVALUATION VALUE 
AND RELIABILITY DATA OF IMAGE DATA 



JUDGE WHETHER PIXEL SHIFT PHOTOGRAPHY 
HAS BEEN PERFORMED NORMALLY, BASED ON 
PIXEL SHIFT EVALUATION VALUE AND 
RELIABILITY DATA OF IMAGE DATA 



DISPLAY JUDGMENT RESULT OF PIXEL SHIFT 
PHOTOGRAPHY ON DISPLAY SCREEN IN 
CAMERA DISPLAY SECTION 



- -S8 
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FIG.3 



PIXEL SHIFT PHOTOGRAPHY HAS 
BEEN PERFORMED NORMALLY 



FIG.4 



SOMETHING IS WRONG IN THE 
PIXEL SHIFT PHOTOGRAPHY 
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FIG 5 A PHOTOGRAPHED IMAGE 1 
(NO SHIFT) 



-r-T-r-T-T 


43- 
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A 





Pir^5 PHOTOGRAPHED IMAGE 2 

riV3 - JD (AFTER SHIFT) 



TARGET IMAGE 



Cir^ CP (SHIFT PHOTOGRAPHED 
riVJ.OO IMAGE 1 TO t 1—0) 



FIG.6A 



PHOTOGRAPHED 
IMAGE 1 



FIG.6B 



PHOTOGRAPHED 
IMAGE 2 
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CTiri "7 A SHIFT PHOTOGRAPHED IMAGE 
rlu./M 2 TO TO -M, CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.45 



FIG.7B 



SHIFT PHOTOGRAPHED IMAGE 
2 TO 4 1 CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.7 



7 ^ SHIFT PHOTOGRAPHED IMAGE 
rlVj./U 2 TO i 2^1, CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.65 



r . n 7n SHIFT PHOTOGRAPHED IMAGE 
h I V3 . / U 2 TO i 3 — M , CORRELATION 



mm 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.4 
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FIG 8 A SHIFT P - HOTOGRAPHED ,MAGE 



2 TO t 0 — 0, CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.7 



FIG 8B SHIFTP . HOTOGRAPH ED L MAGE 



2 TO I 1 -*0, CORRELATION 
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COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.95 



Qp SHIFT PHOTOGRAPHED IMAGE 
^a.OO 2 TO i 2 — 0, CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.9 



F I G 8D SHIFT PHOTOGRAPHED IMAGE 



2 TO 1 3 —0, CORRELATION 



::B:i:;:;:#% 

mm 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.65 
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Plf^ QA SHIFT PHOTOGRAPHED 
riV3.i7/-\ 2TO tO*-1,CORREL/s 



IMAGE 
CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.6 



F\CZ QR SHIFT PHOTOGRAPHED IMAGE 
2 TO t 1 CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.85 



F\CZ QP SHIFT PHOTOGRAPHED IMAGE 
2 TO i 2 CORRELATION 
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COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.8 



FIG 9D SHIFTPHOTOGRAPHED 'MAGE 



2 TO i 3^1, CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.55 
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rirj Hf\A SHIFT PHOTOGRAPHED IMAGE 
llu. I UM 2 TO t 0 *-2. CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1} 0.3 



FIG 1 0 B SHIFT PHOTOGRAPHED IMAGE 



2 TO 1 1 4-2, CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.6 



FIG 1 0C SHIFT PHOTOGRAPjHED IMAGE 



2 TO t 2 <-2, CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.55 



FIG 1 0D SHIFT PHOTOGRAPHED IMAGE 



2 TO t 3 ^2, CORRELATION 



COEFFICIENT (WITH RESPECT TO IMAGE 1) 0.25 
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FIG.11 




DIRECTION (PIXEL) 



FIG. 12 

AMOUNT OF SHIFT IN EACH POSITION (EXAMPLE) 





LEFT 
LOWER 
AREA 



LOWER 
AREA 



RIGHT 
LOWER 
AREA 
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FIG.13A 



PIXEL SHIFT (NORMAL STATE) 




: BEFORE PIXEL SHIFT 
: AFTER PIXEL SHIFT 



FIG.13B 

AMOUNT OF SHIFT IN EACH POSITION (EXAMPLE) 
LEFT UPPER AREA UPPER AREA RIGHT UPPER AREA 



:i.oV^"""i i 1 - 03 !^" i i 1 - 01 L 
i 0.01 i i o.o3 i ; 0.02 



LEFT AREA CENTRAL AREA RIGHT AREA 

:i"i8^""i ; 1.02 1 i : 1 - 01 tJ i 

I 0.01 i i 0.00 i [ 0.02 i 

RIGHT 

LEFT LOWER AREA LOWER AREA LOWER AREA 

[1.00^"": ;i.oV^""j I Jl.OBj 

• 0.01 i i 0.01 i !o.o5 i 



OBLON, SPIVAK, ET AL 
DOCKET #: 215837US2 
INV: Junichi SHINOHARA, et al. 
SHEET JJ_ OF 23 



11/23 



FIG.14A 



PIXEL SHIFT (WHEN THERE IS A HAND MOVEMENT 
OR WHEN THERE IS A FAILURE IN THE MECHANISM) 




: BEFORE PIXEL SHIFT 
: AFTER PIXEL SHIFT 



FIG.14B 

AMOUNT OF SHIFT IN EACH POSITION (EXAMPLE) 
LEFT UPPER AREA UPPER AREA RIGHT UPPER AREA 



[1.19 i i" 1 - 25 L^ i i 1 - 23 L^ ! 

! 0.62 | ; 0.58 ; ! 0.50 | 

LEFT AREA CENTRAL AREA RIGHT AREA 

j i"l.2oj^ I [1. 23|^ i 

! 0.63 ! ; 0.59 ! i 0.59 ! 

RIGHT 

LEFT LOWER AREA LOWER AREA LOWER AREA 

[i.igj""" i I"i.i9 j"" i i" 1 - 20 !" " j 

; 0.59 | | 0.54 j i 0.62 | 
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FIG.15A 

PIXEL SHIFT (WHEN THERE IS A HAND MOVEMENT 
OR WHEN THERE IS A FAILURE IN THE MECHANISM) 



: BEFORE PIXEL SHIFT 
: AFTER PIXEL SHIFT 



FIG.15B 

AMOUNT OF SHIFT IN EACH POSITION (EXAMPLE) 



LEFT UPPER AREA 

! 0.78 : 
LEFT AREA 



UPPER AREA RIGHT UPP, 



: 1.25 



08! 

; 0.03 i 
CENTRAL AREA 



; 0.12! 
RIGHT AREA 



J ^0.12; 


M.02[^ i 




i 0.15 : 


I 0.02 i 


! 0.20 ! 



LEFT LOWER AREA LOWER AREA 

r 0.53 1 r i .01 1 

! 0.04 J*" : 



■tr 



RIGHT 

LOWER AREA 



OBLON, SPIVAK, ET AL 
DOCKET #: 215837US2 
1NV: Junichi SHINOHARA, et al. 
SHEET 13 OF 23 



FIG.16A 

PIXEL SHIFT (MOVING OBJECT EXIST) 




: BEFORE PIXEL SHIFT 
: AFTER PIXEL SHIFT 



FIG.16B 

AMOUNT OF SHIFT IN EACH POSITION (EXAMPLE) 



LEFT UPPER AREA UPPER AREA RIGHT UPPER AREA 



0.01 ! 
LEFT AREA 

i 1 "°" 1 X" 



: 1 .03 



11.01 



r 

0.02 : 



CENTRAL AREA RIGHT AREA 



0.01 



LEFT LOWER AREA LOWER AREA 



\ I 1 - 21 




J 1.08 


! 1.63 




3.21 



RIGHT 

LOWER AREA 



i 1 - 00 t 


1*1.01 £ | 


: J 1 - 08 ! 


I o.oi i 


■ o.oi i 


1 0.06 : 
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FIG.17 

SECOND EMBODIMENT 



IS THERE AN 
INSTRUSTION TO START PIXEL SHIFT 
PHOTOGRAPHING? 



JUDGE A FAILURE OF PIXEL SHIFT UNIT BASED ON PIXEL SHIFT RELATIONAL 
DATA OF THE LAST SEVERAL TIMES STORED IN MEMORY GROUP 



IS THERE A FAILURE OF 
PIXEL SHIFT UNIT? 



DISPLAY THE FAILURE OF PIXEL 
SHIFT UNIT IN CAMERA 
INSTRUCTION SECTION 



IS IT NECESSARY TO CHANGE 
ADJUSTMENT VALUE? 



CALCULATE NEW ADJUSTMENT VALUE AND CHANGE ADJUSTMENT 
VALUE 



PERFORM FIRST PHOTOGRAPHING AND STORE IMAGE 
DATA IN MEMORY GROUP 



SET ADJUSTMENT VALUE IN PIXEL SHIFT DRIVE CIRCUIT TO SHIFT 
IMAGE SENSING DEVICE BY ONE PIXEL IN LONGITUDINAL DIRECTION 



PERFORM SECOND PHOTOGRAPHING, WITH IMAGE SENSING DEVICE 
SHIFTED BY ONE PIXEL IN LONGITUDINAL DIRECTION, AND STORE 
IMAGE DATA IN MEMORY GROUP 



COMBINE FIRST AND SECOND IMAGES STORED IN MEMORY GROUP 



CALCULATE PIXEL SHIFT EVALUATION VALUE AND RELIABILITY 
DATA OF IMAGE DATA, AND STORE THESE IN MEMORY GROUP AS 
PIXEL SHIFT RELATIONAL DATA 



JUDGE WHETHER PIXEL SHIFT PHOTOGRAPHY HAS BEEN 
PERFORMED NORMALLY, BASED ON PIXEL SHIFT EVALUATION VALUE f S21 
AND RELIABILITY DATA OF IMAGE DATA 



DISPLAY JUDGMENT RESULT OF PIXEL SHIFT PHOTOGRAPHY ON 
DISPLAY SCREEN IN CAMERA DISPLAY SECTION 



( END 
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FIG.18 



THE PIXEL SHIFT UNIT 
HAS A FAILURE 



OBLON, SPIVAK, ET AL 
DOCKETS 215837US2 
INV: Junichi SHINOHARA, et al. 
SHEET ___ OF__23_ 



FIG.19 

THIRD EMBODIMENT 




EXECUTE SELF CALIBRATION MODE TO CALCULATE OPTIMUM j 
tAt ADJUSTMENT VALUE | 




■^m 1 CO 

PERFORM FIRST PHOTOGRA PHING AND STORE IMAGE DATA IN MEMORY JL s36 
GROUP ——I 



DIRECTION 



PFRFQRM SECOND PHOTO GRAPHING, WITH IMAGE SENSING DEVICE 

SHIFTED BY ONE PKELIN LONGITUDINAL DIRECTION, AND STORE IMAGE + 
^ n DATA IN MEMORY GROUP 



COMBINE FIRST AND SECOND IMAGES STORED IN MEMORY GROUP j s39 

CALCULATE PIXEL SHIFT EVAL UATION VALUE AND RELIABILITY UAI A l s40 
OF IMAGE DATA J 



ii .~ nc va/i-iftmfr PIXEL SHIFT PH OTOGRAPHY HAS BEEN PERFORMED 
JUD NORMaIlY BASE E D ON PIXEL SHIFT EVALUATION VALUE AND 
NU ' RELIABILITY DATA OF IMAGE DATA 



DISPLAY JUDGMENT RESULT O F PIXEL SHIFT ' PHOTUUff PHY ON 
DISPLAY SCREEN IN CAMERA DISPLAY SECTION 



END 
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FIG.20A FIG.20B 



PHOTOGRAPHED IMAGE 1 TARGET IMAGE 

(NO SHIFT) (IMAGE 1 I 1 —0) 
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FIG.21A 



PHOTOGRAPHED IMAGE 2, 
(ADJUSTMENT VALUE = 80) 




COINCIDENCE DEGREE WITH 
RESPECT TO TARGET IMAGE = 0.75 



FIG.21B 



PHOTOGRAPHED IMAGE 3, 
(ADJUSTMENT VALUE = 90) 



COINCIDENCE DEGREE WITH 
RESPECT TO TARGET IMAGE = 0.85 



FIG.21C 

PHOTOGRAPHED IMAGE 4, 
(ADJUSTMENT VALUE = 100) 




COINCIDENCE DEGREE WITH 
RESPECT TO TARGET IMAGE = 0.95 



FIG.21D 

PHOTOGRAPHED IMAGE 5, 
(ADJUSTMENT VALUE = 110) 



COINCIDENCE DEGREE WITH 
RESPECT TO TARGET IMAGE = 0.93 



FIG.21E 



PHOTOGRAPHED IMAGE 6, 
(ADJUSTMENT VALUE = 120) 




COINCIDENCE DEGREE WITH 
RESPECT TO TARGET IMAGE = 0.83 



FIG.21F 



PHOTOGRAPHED IMAGE 7, 
(ADJUSTMENT VALUE = 130) 



COINCIDENCE DEGREE WITH 
RESPECT TO TARGET IMAGE = 0.73 



FIG.21G 



PHOTOGRAPHED IMAGE 8, 
(ADJUSTMENT VALUE = 140) 



FIG.21H 

PHOTOGRAPHED IMAGE 9, 
(ADJUSTMENT VALUE = 150) 



COINCIDENCE DEGREE WITH 
RESPECT TO TARGET IMAGE = 0.63 



COINCIDENCE DEGREE WITH 
RESPECT TO TARGET IMAGE = 0.53 
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FIG.22 

ADJUSTMENT VALUE AND COINCIDENCE DEGREE 
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FIG. 23 



(a) IN THE CASE OF COMPARATIVELY GOOD 
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0.97 PIXEL WITH 
ADJUSTMENT 
VALUE OF 100 



zs 



-A 



1.25 PIXEL WITH 
ADJUSTMENT 
VALUE OF 130 



INITIAL ADJUSTMENT 
VALUE = 115 



110 120 130 140 150 
\ ADJUSTMENT VALUE 



OPTIMUM ADJUSTMENT 
VALUE = 103 
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FIG.24 



(b) IN THE CASE OF ADJUSTMENT VALUE 
IS DEVIATED LARGELY 
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0.93 PIXEL WITH] 
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FIG.25 
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(2) IN THE CASE OF ADJUSTMENT 
VALUE ATONE POINT 




U LINE SHOWING 
AVERAGE 
ADJUSTMENT VALUE 
., VS AMOUNT OF 
J-v SHIFT 



_L_ 



INITIAL ADJUSTMENT 
VALUE = 115 



110 120 130 140 150 
\ ADJUSTMENT VALUE 



OPTIMUM ADJUSTMENT 
VALUE = 103 
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FIG.26 



FOURTH EMBODIMENT 




PERFORM FIRST 
PHOTOGRAPHING AND 
STORE IMAGE DATA IN 

MEMORY GROUP 



SHIFT IMAGE SENSING 
DEVICE BY ONE PIXEL 
IN LONGITUDINAL 
DIRECTION 



▼ < as 

PERFORM SECOND 
PHOTOGRAPHING, 
WITH IMAGE SENSING 
DEVICE SHIFTED BY 
ONE PIXEL IN 
LONGITUDINAL 
DIRECTION, AND 
STORE IMAGE DATA IN 
MEMORY GROUP 



COMBINE FIRST AND 

SECOND IMAGES 
STORED IN MEMORY 
GROUP 

CALCULATE PIXEL 
SHIFT EVALUATION 
VALUE AND 
RELIABILITY DATA OF 
IMAGE DATA 



S57 



JUDGE WHETHER 
PIXEL SHIFT 
PHOTOGRAPHIY HAS 
BEEN PERFORMED 
NORMALLY, BASED 

ON PIXEL SHIFT 
EVALUATION VALUE 

AND RELIABILITY 
DATA OF IMAGE DATA 

* ! 558 

DISPLAY JUDGMENT 

RESULT OF PIXEL 
SHIFT PHOTOGRAPHY 
ON DISPLAY SCREEN 
IN CAMERA DISPLAY 
SECTION 



SHIFT IMAGE SENSING DEVICE BY ONE PIXEL IN 
LONGITUDINAL DIRECTION 



SET ADJUSTMENT VALUE IN PIXEL SHIFT DRIVE CIRCUIT 
PERFORM SECOND PHOTOGRAPHING, WITH IMAGE SENSING 
DEVICE SHIFTED BY ONE PIXEL IN LONGITUDINAL 
DIRECTION, AND STORE IMAGE DATA IN MEMORY GROUP 




PERFORM SECOND PHOTOGRAPHING, WITH IMAGE 
SENSING DEVICE SHIFTED BY ONE PIXEL IN LONGITUDINAL \ S70 
DIRECTION, AND STORE IMAGE DATA IN MEMORY GROUP 



CALCULATE ADJUSTMENT VALUE BASED ON PIXEL SHIFT 
EVALUATION VALUE AND RELIABILITY DATA OF IMAGE DATA ' 



JUDGE WHETHER PIXEL SHIFT PHOTOGRAPHY HAS BEEN 
PERFORMED NORMALLY, BASED ON PIXEL SHIFT 
EVALUATION VALUE AND RELIABILITY DATA OF IMAGE DATA 



DISPLAY JUDGMENT RESULT OF PIXEL SHIFT PHOTOGRAPHY 

ON DISPLAY SCREEN IN CAMERA DISPLAY SECTION T S74 
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FIG.27A 

CCD SHIFT METHOD 
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FIG.27B 

LPF GRADIENT METHOD 













7 


SENSING 
'ICE 












3 




-ILJLU 

< 
2 





FIG.27C 

LENS SHIFT METHOD 
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